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Artificial Intelligence and National Security 


Summary 


Artificial intelligence (AI) is a rapidly growing field of technology with potentially significant 
implications for national security. As such, the United States and other nations are developing AI 
applications for a range of military functions. AI research is underway in the fields of intelligence 
collection and analysis, logistics, cyber operations, information operations, command and control, 
and in a variety of semiautonomous and autonomous vehicles. Already, AI has been incorporated 
into military operations in Iraq and Syria. Congressional action has the potential to shape the 
technology’s development further, with budgetary and legislative decisions influencing the 
growth of military applications as well as the pace of their adoption. 


AI technologies present unique challenges for military integration, particularly because the bulk 
of AI development is happening in the commercial sector. Although AI is not unique in this 
regard, the defense acquisition process may need to be adapted for acquiring emerging 
technologies like AI. In addition, many commercial AI applications must undergo significant 
modification prior to being functional for the military. A number of cultural issues also challenge 
AI acquisition, as some commercial AI companies are averse to partnering with the Department 
of Defense (DOD) due to ethical concerns, and even within the department, there can be 
resistance to incorporating AI technology into existing weapons systems and processes. 


Potential international rivals in the AI market are creating pressure for the United States to 
compete for innovative military AI applications. China is a leading competitor in this regard, 
releasing a plan in 2017 to capture the global lead in AI development by 2030. Currently, China is 
primarily focused on using AI to make faster and more well-informed decisions, as well as on 
developing a variety of autonomous military vehicles. Russia is also active in military AI 
development, with a primary focus on robotics. 


Although AI has the potential to impart a number of advantages in the military context, it may 
also introduce distinct challenges. AI technology could, for example, facilitate autonomous 
operations, lead to more informed military decisionmaking, and increase the speed and scale of 
military action. However, it may also be unpredictable or vulnerable to unique forms of 
manipulation. As a result of these factors, analysts hold a broad range of opinions on how 
influential AI will be in future combat operations. While a small number of analysts believe that 
the technology will have minimal impact, most believe that AI will have at least an 
evolutionary—if not revolutionary—effect. 


Military AI development presents a number of potential issues for Congress: 
e What is the right balance of commercial and government funding for AI 
development? 


e How might Congress influence defense acquisition reform initiatives that 
facilitate military AI development? 


e What changes, if any, are necessary in Congress and DOD to implement effective 
oversight of AI development? 


e How should the United States balance research and development related to 
artificial intelligence and autonomous systems with ethical considerations? 


e What legislative or regulatory changes are necessary for the integration of 
military AI applications? 


e What measures can Congress take to help manage the AI competition globally? 


Congressional Research Service 


Artificial Intelligence and National Security 


Contents 
HiME OCU C LIT sce ce cares ecataeac te T AE A E E E E 1 
Al Terminology and Background ..0...... cee eceeceseeseeeeceseeseeeeceaecaeeeceesecaaeeaeeseceaeeaeeseeaecaeeeneeaecaaeeaeeaees 1 
Issues Tor’ COMBTESS vsscgsis 14 dasase, scbesacSeesvecs yacavanenyeessedorstiassdipatvoneosastivshqeseatssenteansteasentorseesbantenesasa EEAS 5 
ALApplications for Defense snenie e e a a A 9 
Intelligence, Surveillance, and Reconnaissance.........ccccccccsceessecseeeeceseceseceseceseceeeeeeeeeeeesneeaes 10 
TSO GISTICS AE E AE saencast sensu Deanncetessateceeassinsenseatessdsentivesdaaterentosnedtes 11 
Cyberspace Operations ic. 225 srsusecesexsdscceendesesvarsanhdauienss aea NEREAREN 11 
Information Operations and “Deep Fakes” .......cseecesssseccesecseeeeeesececeeeeeeecaeeaeeeeeeaeeaeeeneeaeenee 12 
Cömmand and Control vis: oss stesenscecvessassest iaren ie sa-vosed aaah e a ar e iaee aerei 12 
Semiautonomous and Autonomous Vehicles .........seeseseeeeseesesesereeessseseesrssrstsresrsssrsesenseseeersenss 13 
Lethal Autonomous Weapon Systems (LAWS) .........cssscssccsssseeeseeeeceeeeeeeseceaeeaeeeeceaecaeeeneeaeuee 15 
Military AI Integration Challenges... ce ceeeessssceeeeeeeceseeseeecesecaeeeeeesecseeeneesecsaesaeeeeeaeeaeeeneeaeenee 16 
TECHNO lO EY iee cece Saeed etait se EA A a EE O acest E E EEAS 17 
PROCESS sucesora aeriene RT OENE EEE EEEE EE EE EEEE EEE S 17 
Personnel nreresi ienesa isee a i A E A RRA TEE 19 
(IREI vs vcs e E E E E TA A E E EEE 19 
International Competitors siissssrsssusisenevisenoresisdsnsrissisneto rnarnodsteses e sasidan een ia eisa eonun s naoi Eia 20 
CVAD sx ss5 sis Sevss San ia cesau devas Saale banecethasseadacnsvades asad’ cesteescev eas dea desasnadsanae asst 21 
RUSSA irecte oiee E E AEEA A E EE E E E E 25 
Tnternatióonal Institutions... .ssice..scdisess sieves. sveendiateessentedieschaciveneodedeaegoseivesceandonsdostieneanseteneeaaionegencaats 27 
AI Opportunities and Challenges .0....... cc eceecceseeseesecesecseeeeeeseceaeeaeesecsaeeaeeseceaecaeeeeeeaecsaeeaeeaeeaeeatees 27 
AULOMOMY AE TEE 28 
Speed aiid Endüränte nenesie a a E E E E a TE EEn 29 
SCALING N S O TE T A EAEE 29 
Information Superiority... eee eseeccesecscesseeeececeeseeeecaecseeeeceaecsaeeceesecsaeeaeeseceaeeaeseeeeaecaeeeneeaeenee 29 
Predictability 2r..c2cesstsscextsestossessstoosaeesidceshasteseaventuttessutetsdtousedassnelaceasageeesnssadextisnseiststasleciaaapennis 30 
XP lama ait Y estaire eenst See eLo er eaen EEEE A OE Aeee e EEEE atta ENEE AE ATOE ET 32 
ExploitaiOfnonienennine ninnan aa e R a a a a a 34 
AI’s Potential Impact on Combat... eceeccesesseeeecesecseeeseeseceaeeseeseceaeeaeeeceaecaeeeeeeaeceaeeaeeeeeaeeatees 35 
Minimal Impact: Om: Combatissa aa cveusosacsnesnacienianaateareancens 35 
Evolutionary Impact on Combat ...........ccecescceseeseeseceseeseeeecesecaeeeneeseceaeeaeeseceaesaeeeeceaecaeeeneeaeenee 36 
Revolutionary Impact on Combat ........ cc ecescesesseeseceseeeeeecesecseeeeeesecaaeeaeeaeceaeeaeeeeeeaeeaeeeneeaeenee 37 
Figures 
Figure 1. Relationships of Selected AI Definitions .0...... eee eeceseesceeeceseeeeeseesecseeeeeesecaeeaeeneaeeaeees 4 
Figure 2. Chinese Investment in U.S. AI Companies, 2010-201 7........cecesssceseeseeeeceeeneeeneeseeeee 23 
Figure 3. Value of Autonomy to DOD Missions ..............esesceeseeseeseeeeeeseceeeeeeeseceaeeaeeecaecaeeeneeaeenee 28 
Figure 4. AI and Image Classifying Errors..........cceceseescesseseeeecesecseeeceesecaeeeneeseceaeeaeeeceaeeaeeeneeaeenee 31 
Figùre S.A Land Context aiteis neroni rea a raaa rarai eeN 31 
Figúre 6. Adversarial Images. .i.cciescstssesssscsecssesieruansdsntassancatsdsoacactabnasaddesaazsdennendeseiondssevssscesenesasuves 34 


Congressional Research Service 


Artificial Intelligence and National Security 


Tables 

Table 1. Taxonomy of Historical AI Definitions 00.0.0... cee cecceseceseceseceeceseeeeeeeeeeeeeeeeaeeeaeeeaeeeaeenaees 3 
Contacts 

Author InfOriniati On ?. 52 ssc sd sdes sesccceusudesdh ove sucessuebadessnatecdessuzcavie’ewabu'cechnebedesuna tendon sdeatvlavateddsaneladesanasudss 39 
AckiowledgmMentS ieioea aeaieie EEE EE caus oad etyanez EES OaE S aiaa 39 


Congressional Research Service 


Artificial Intelligence and National Security 


Introduction! 


Artificial intelligence (AI) is a rapidly growing field of technology that is capturing the attention 
of commercial investors, defense intellectuals, policymakers, and international competitors alike, 
as evidenced by a number of recent initiatives. On July 20, 2017, the Chinese government 
released a strategy detailing its plan to take the lead in AI by 2030. Less than two months later 
Vladimir Putin publicly announced Russia’s intent to pursue AI technologies, stating, “[W]hoever 
becomes the leader in this field will rule the world.”” Similarly, the U.S. National Defense 
Strategy, released in January 2018, identified artificial intelligence as one of the key technologies 
that will “ensure [the United States] will be able to fight and win the wars of the future.”? 


The U.S. military is already integrating AI systems into combat via a spearhead initiative called 
Project Maven, which has used AI algorithms to identify insurgent targets in Iraq and Syria.* 
These dynamics raise several questions that have been addressed in congressional hearings: What 
types of military AI applications are possible, and what limits, if any, should be imposed? What 
unique advantages and vulnerabilities come with employing AI for defense? How will AI change 
warfare, and what influence will it have on the military balance with U.S. competitors? Congress 
has a number of oversight, budgetary, and legislative tools available that it may use to influence 
the answers to these questions and shape the future development of AI technology. 


AI Terminology and Background’ 


Almost all academic studies in artificial intelligence acknowledge that no commonly accepted 
definition of AI exists, in part because of the diverse approaches to research in the field. 
Likewise, although Section 238 of the FY2019 National Defense Authorization Act (NDAA) 
directs the Secretary of Defense to produce a definition of artificial intelligence by August 13, 
2019, no official U.S. government definition of AI yet exists. The FY2019 NDAA does, 
however, provide a definition of AI for the purposes of Section 238: 


1. Any artificial system that performs tasks under varying and unpredictable 
circumstances without significant human oversight, or that can learn from 
experience and improve performance when exposed to data sets. 


2. An artificial system developed in computer software, physical hardware, or other 
context that solves tasks requiring human-like perception, cognition, planning, 
learning, communication, or physical action. 


' This report was originally written by Daniel S. Hoadley, U.S. Air Force Fellow. It has been updated by Kelley M. 
Sayler, Analyst in Advanced Technology and Global Security. 

2 China State Council, “A Next Generation Artificial Intelligence Development Plan,” July 20, 2017, translated by New 
America, https://www.newamerica.org/documents/1959/translation-fulltext-8.1.17.pdf, and Tom Simonite, “For 
Superpowers, Artificial Intelligence Fuels New Global Arms Race,” Wired, August 8, 2017, https://www.wired.com/ 
story/for-superpowers-artificial-intelligence-fuels-new-global-arms-race. 

3 Department of Defense, Summary of the 2018 National Defense Strategy, p.3, https://dod.defense.gov/Portals/1/ 
Documents/pubs/2018-National-Defense-Strategy-Summary.pdf. 

4 Marcus Weisgerber, “The Pentagon’s New Algorithmic Warfare Cell Gets Its First Mission: Hunt ISIS,” Defense 
One, May 14, 2017, http://www.defenseone.com/technology/2017/05/pentagons-new-algorithmic-warfare-cell-gets-its- 
first-mission-hunt-isis/137833/. 

5 For a general overview of AI, see CRS In Focus IF10608, Overview of Artificial Intelligence, by Laurie A. Harris. 

é P.L. 115-232, Section 2, Division A, Title IT, §238. 
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3. An artificial system designed to think or act like a human, including cognitive 
architectures and neural networks. 


4. A set of techniques, including machine learning that is designed to approximate a 
cognitive task. 


5. An artificial system designed to act rationally, including an intelligent software 
agent or embodied robot that achieves goals using perception, planning, 
reasoning, learning, communicating, decision-making, and acting.’ 


This definition encompasses many of the descriptions in Table 1 below, which summarizes 
various AI definitions in academic literature. 


The field of AI research began in the 1940s, but an explosion of interest in AI began around 2010 
due to the convergence of three enabling developments: (1) the availability of “big data” sources, 
(2) improvements to machine learning approaches, and (3) increases in computer processing 
power.® This growth has advanced the state of Narrow AI, which refers to algorithms that address 
specific problem sets like game playing, image recognition, and navigation. All current AI 
systems fall into the Narrow AI category. The most prevalent approach to Narrow AI is machine 
learning, which involves statistical algorithms that replicate human cognitive tasks by deriving 
their own procedures through analysis of large training data sets.” During the training process, the 
computer system creates its own statistical model to accomplish the specified task in situations it 
has not previously encountered. 


Experts generally agree that it will be many decades before the field advances to develop General 
AI, which refers to systems capable of human-level intelligence across a broad range of tasks.!° 
Nevertheless, the rapid advancements in Narrow AI have sparked a wave of investment, with U.S. 
venture capitalists raising an estimated $18.5 billion for AI research in 2019 alone.!! Similarly, 
DOD’s unclassified investments in AI have grown from just over $600 million in FY2016 to $2.5 
billion in FY2021 (including investments in autonomy), with the Department reportedly 
maintaining over 600 active AI projects.’ 


7 Tbid. For a critique of this definition, see Defense Innovation Board (DIB), “AI Principles: Recommendations on the 
Ethical Use of Artificial Intelligence by the Department of Defense — Supporting Document,” November 2019, pp. 8- 
10, https://media.defense.gov/2019/Oct/3 1/2002204459/- 1/-1/0/ 
DIB_AI_PRINCIPLES_SUPPORTING_DOCUMENT.PDEF. The DIB instead defines artificial intelligence as “a 
variety of information processing techniques and technologies used to perform a goal-oriented task and the means to 
reason in pursuit of that task.” 


8 Executive Office of the President, National Science and Technology Council, Committee on Technology, Preparing 
for the Future of Artificial Intelligence, October 12, 2016, p. 6, https://obamawhitehouse.archives.gov/sites/default/ 
files/whitehouse_files/microsites/ostp/NSTC/preparing_for_the_future_of_ai-.pdf. 


° For additional information about machine learning algorithms, see Greg Allen, Understanding AI Technology: A 
concise, practical, and readable overview of Artificial Intelligence and Machine Learning technology designed for non- 
technical managers, officers, and executives, Joint Artificial Intelligence Center, April 2020, 
https://www.ai.mil/docs/Understanding %20AI%20Technology. pdf. 


10 Tbid., pp. 7-9. 


11 Chris O’Brien, “AI startups raised $18.5 billion in 2019, setting new funding record,” Venture Beat, January 14, 
2020, https://venturebeat.com/2020/01/14/ai-startups-raised-18-5-billion-in-2019-setting-new-funding-record/. 


12 See Govini, Department of Defense Artificial Intelligence, Big Data, and Cloud Taxonomy, December 3, 2017, p. 9; 
Office of the Under Secretary of Defense (Comptroller)/Chief Financial Officer, Defense Budget Overview: United 
States Department of Defense FY2021 Budget Request, February 2020, p. 1-9; and Brendan McCord, Eye on AI, 
August 28, 2019, transcript available at https://staticl .squarespace.com/static/5b75ac0285ede1b470f58ae2/t/ 
5d6aa8edb9 1b0c0001c7a05f/1567. 
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AI has a number of unique characteristics that may be important to consider as these technologies 
enter the national security arena. First, AI has the potential to be integrated across a variety of 
applications, improving the so-called “Internet of Things” in which disparate devices are 
networked together to optimize performance.'* As Kevin Kelley, the founder of Wired magazine, 
states, “[AI] will enliven inert objects, much as electricity did more than a century ago. 
Everything that we formerly electrified we will now cognitize.”'* Second, many AI applications 
are dual-use, meaning they have both military and civil applications. For example, image 
recognition algorithms can be trained to recognize cats in YouTube videos as well as terrorist 
activity in full motion video captured by uninhabited aerial vehicles over Syria or Afghanistan." 
Third, AI is a relatively transparent enabling capability, meaning that its integration into a product 
may not be immediately recognizable. By and large, AI procurement will not result in countable 
objects. Rather, the algorithm will be purchased separately and incorporated into a larger system. 
As an expert in the field points out, “We will not buy AI. It will be used to solve problems, and 
there will be an expectation that AI will be infused in most things we do.”!6 


Al Concepts 


Table |.Taxonomy of Historical Al Definitions 


Systems That Think Like Humans Systems That Think Rationally 


“The automation of activities that we associate with “The study of computations that make possible to 
human thinking, activities such as decision making, perceive, reason, and act.” 

roblem solving, and learning.” z 
P 8 8 —Winston, 1992 


—Bellman, 1978 


Systems That Act Like Humans Systems That Act Rationally 


“The art of creating machines that perform functions “The branch of computer science that is concerned 
that require intelligence when performed by people.” | with the automation of intelligent behavior.” 


—Kurzweil, 1990 —Luger and Stubblefield, 1993 


Selected Definitions—Where possible, an official U.S. government document is cited. 


e Automated systems. “A physical system that functions with no (or limited) human operator involvement, 
typically in structured and unchanging environments, and the system’s performance is limited to the specific 
set of actions that it has been designed to accomplish ... typically these are well-defined tasks that have 
predetermined responses according to simple scripted or rule-based prescriptions.” !7 


Autonomy. “The condition or quality of being self-governing in order to achieve an assigned task based on 
the system’s own situational awareness (integrated sensing, perceiving, and analyzing), planning, and decision 
making.”!8 


13 See Steve Ranger, “What is the IoT? Everything you need to know about the Internet of Things right now,” 
ZDNet.com, August 21, 2018, https://www.zdnet.com/article/what-is-the-internet-of-things-everything-you-need-to- 
know-about-the-iot-right-now/. 


14 Kevin Kelly, “The Three Breakthroughs That Have Finally Unleashed AI on the World,” Wired, October 27, 2014, 
https://www.wired.com/2014/10/future-of-artificial-intelligence. 


15 Greg Allen and Taniel Chan, Artificial Intelligence and National Security, Belfer Center for Science and 
International Affairs, July 2017, p. 47. 


16 Steve Mills, Presentation at the Global Security Forum, Center for Strategic and International Studies, Washington, 
DC, November 7, 2017. 


17 Andrew Ilachinski, AI, Robots, and Swarms: Issues, Questions, and Recommended Studies, Center for Naval 
Analysis, January 2017, p. 6. 


'8 Department of Defense, Joint Concept for Robotic and Autonomous Systems, October 19, 2016, p. A-3. 
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Autonomous Weapon System (aka Lethal Autonomous Weapon System, LAWS). “A weapon system 
that, once activated, can select and engage targets without further intervention by a human operator.”!9 


Human-Supervised Autonomous Weapon System. “An autonomous weapon system that is designed to 
provide human operators with the ability to intervene and terminate engagements, including in the event 
of a weapon system failure, before unacceptable levels of damage occur.”20 


Semi-Autonomous Weapon System. “A weapon system that, once activated, is intended to only engage 
individual targets or specific target groups that have been selected by a human operator.””?! 


Robot. “A powered machine capable of executing a set of actions by direct human control, computer 
control, or a combination of both. At a minimum it is comprised of a platform, software, and a power 
source.”’22 


Understanding the relationships between these terms can be challenging, as they may be used interchangeably in 
the literature and definitions often conflict with one another. For example, some studies delineate between 
automated systems and autonomous systems based on the system’s complexity, arguing that automated systems 
are strictly rule-based, while autonomous systems exhibit artificial intelligence. Some, including the Department of 
Defense, categorize autonomous weapon systems based not on the system’s complexity, but rather on the type of 
function being executed without human intervention (e.g., target selection and engagement).?3 Still others describe 
Al as a means of automating cognitive tasks, with robotics automating physical tasks. This framework, however, 
may not be sufficient to describe how Al systems function, as such systems do not merely replicate human 
cognitive functions and often produce unanticipated outputs. In addition, a robot may be automated or 
autonomous and may or may not contain an Al algorithm. Figure I illustrates these relationships, based on the 
above selected definitions of each term. 


Figure |. Relationships of Selected Al Definitions 


Source: CRS. 


' Department of Defense, Directive 3000.09, Autonomy in Weapon Systems, http://www.esd.whs.mil/Portals/54/ 
Documents/DD/issuances/DODd/300009p.pdf. 


20 Thid. 
21 Tid. 
22 Department of Defense, Joint Concept for Robotic and Autonomous Systems, p. A-3. 


23 See Paul Scharre and Michael C. Horowitz, An Introduction to Autonomy in Weapon Systems, Center for a New 
American Security, February 2015, pp. 6-7. 


Congressional Research Service 4 


Artificial Intelligence and National Security 


Issues for Congress 


A number of Members of Congress have called for action on military AI. During the opening 
comments to a January 2018 hearing before the House Armed Services Subcommittee on 
Emerging Threats, the subcommittee chair called for a “national level effort” to preserve a 
technological edge in the field of AI.” Former Deputy Secretary of Defense Robert Work argued 
in a November 2017 interview that the federal government needs to address AI issues at the 
highest levels, further stating that “this is not something the Pentagon can fix by itself.” 


For its part, DOD has published a classified AI strategy and is carrying out multiple tasks directed 
by DOD guidance and the FY2019 and FY2020 NDAAs, including 


e establishing a Joint Artificial Intelligence Center (JAIC, pronounced “jake’’), 
which now “coordinate[s] the efforts of the Department to develop, mature, and 


transition artificial intelligence technologies into operational use”; 


e publishing a strategic roadmap for AI development and fielding, as well as 
guidance on “appropriate ethical, legal, and other policies for the Department 
governing the development and use of artificial intelligence enabled systems and 


technologies in operational situations”;”’ 


e establishing a National Security Commission on Artificial Intelligence to conduct 
a comprehensive assessment of militarily relevant AI technologies and provide 
recommendations for strengthening U.S. competitiveness;”* and 


24 U.S. Congress, House of Representatives Committee on Armed Services, Subcommittee on Emerging Threats and 
Capabilities, Hearing on China’s Pursuit of Emerging Technologies, 115" Cong., 2" sess., January 9, 2018, transcript 
available at http://www.cq.com/doc/congressionaltranscripts-5244793?1; remarks by Rep. Joe Wilson. 


25 Colin Clark, “Our Artificial Intelligence ‘Sputnik Moment’ is Now: Eric Schmidt and Bob Work,” Breaking 
Defense, November 1, 2017, https://breakingdefense.com/2017/1 1/our-artificial-intelligence-sputnik-moment-is-now- 
eric-schmidt-bob-work/. 


26 Sydney J. Freedberg, Jr., “Pentagon Rolls Out Major Cyber, AI Strategies This Summer,” Breaking Defense, July 17, 
2018, https://breakingdefense.com/2018/07/pentagon-rolls-out-major-cyber-ai-strategies-this-summer/; and P.L. 115- 
232, Section 2, Division A, Title X, §1051. 


27 P.L. 115-232, Section 2, Division A, Title II, §238. In support of this mandate, the Defense Innovation Board, an 
independent federal advisory committee to the Secretary of Defense, drafted recommendations for the ethical use of 
artificial intelligence. See Defense Innovation Board, “AI Principles: Recommendations on the Ethical Use of Artificial 
Intelligence by the Department of Defense,” October 31, 2019, https://media.defense.gow/2019/Oct/3 1/2002204458/- 1/- 
1/0/DIB_AI_PRINCIPLES_PRIMARY_DOCUMENT.PDF. RAND Corporation, a federally funded research and 
development center, additionally conducted a review of DOD posture for AI. See Danielle C. Tarraf et al., The 
Department of Defense Posture for Artificial Intelligence: Assessment and Recommendations, RAND Corporation, 
2019, https://www.rand.org/pubs/research_reports/RR4229. html. 


28 Ibid. and P.L. 115-232, Section 2, Division A, Title X, §1051. The Commission’s Interim Report, which assesses the 
challenges and opportunities of militarily relevant AI technologies, is available at https://drive.google.com/file/d/ 
153OrxnuGEjsUvlxWsFYauslwNeCEkvUb/view. It identifies five key lines of effort for U.S. AI competitiveness: (1) 
investing in research and development, (2) applying AI to national security missions, (3) training and recruiting AI 
talent, (4) protecting and building upon U.S. technology advantages, and (5) marshalling global AI cooperation. The 
commission is releasing recommendations for implementing these lines of effort every quarter. Its first quarter 
recommendations, released in March 2020, are available at 
https://drive.google.com/file/d/1wkPh8GbS5drBrKBg6OhGuS5oNaTEERbKss/view. Second quarter recommendations, 
released in July 2020, are available at 

https://drive.google.com/file/d/1 LDrd6T7H50ry9uX NA6cwhsrtnpQ63EWH/view. Third quarter recommendations, 
released in October 2020, are available at https://drive. google.com/file/d/1jg9 YINagGI_Orid- HX Y-fvJO AejlFliy/view. 
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e developing an artificial intelligence education strategy for servicemembers.” 
These initiatives will present a number of oversight opportunities for Congress. 


In addition, Congress may consider the adequacy of current DOD funding levels for AI. 
Lieutenant General John Shanahan, the former director of the JAIC, identified funding as a 
barrier to future progress.*° Although DOD funding for AI has increased, beginning in 2018—to 
include the JAIC’s $1.75 billion six-year budget and the Defense Advanced Research Projects 
Agency’s (DARPA’s) $2 billion multiyear investment in over 20 AI programs—some experts 
have argued that additional DOD funding will be required to keep pace with U.S. competitors and 
avoid an “innovation deficit” in military technology.*! 


Critics of increased federal funding contend that significant increases to appropriations may not 
be required, as the military should be leveraging research and development (R&D) conducted in 
the commercial sector. The 2017 National Security Strategy identifies a need to “establish 
strategic partnerships to align private sector R&D resources to priority national security 
applications” and to reward government agencies that “take risks and rapidly field emerging 
commercial technologies.” In addition, the Office of Management and Budget directed DOD in 
preparing its FY 2020 budget to “seek to rapidly field innovative technologies from the private 
sector, where possible, that are easily adaptable to Federal needs, rather than reinventing solutions 
in parallel.” Some experts in the national security community also argue that it would not be a 
responsible use of taxpayer money to duplicate efforts devoted to AI R&D in the commercial 
sector when companies take products 90% of the way to a useable military application.** Others 
contend that a number of barriers stand in the way of transitioning AI commercial technology to 
DOD, and that reforming aspects of the defense acquisition process may be necessary.” These 
issues are discussed in more detail later in this report.*° 


One impediment to accurately evaluating funding levels for AI is the lack of a stand-alone AI 
Program Element (PE) in DOD funding tables. As a result, AI R&D appropriations are spread 
throughout generally titled PEs and incorporated into funding for larger systems with AI 


2 P.L. 116-92, Section 2, Division A, Title II, §256. 


30 Justin Doubleday, “Project Maven Aims to Introduce AI tools into Services’ Intel Systems,” Inside Defense, January 
5, 2018, https://insidedefense.com/inside-army/project-maven-aims-introduce-ai-tools-services-intel-systems, and 
Jason Sherman, “ASB: S&T Funding Inadequate to Support “Big Bets’ on Disruptive Technologies,” Inside Defense, 
December 15, 2017, https://insidedefense.com/inside-army/asb-st-funding-inadequate-support-big-bets-disruptive- 
technologies. 


31 “DARPA Announces $2 Billion Campaign to Develop Next Wave of AI Technologies,” DARPA, September 7, 
2018, https://www.darpa.mil/news-events/2018-09-07; and Elsa B. Kania, “Battlefield Singularity: Artificial 
Intelligence, Military Revolution, and China’s Future Military Power,” Center for a New American Security, 
November 28, 2017, pp. 40-41, https://s3.amazonaws.com/files.cnas.org/documents/Battlefield-Singularity-November- 
2017.pdf?mtime=20171129235804. 

32 The White House, National Security Strategy of the United States of America, December 2017, p. 21, 
https://www.whitehouse. gov/wp-content/uploads/2017/12/NSS-Final-12-18-2017-0905-2.pdf. 


33 Executive Office of the President, Director, Office of Management and Budget, Memorandum for the Heads of 
Executive Departments and Agencies, “FY 2020 Administration Research and Development Budget Priorities,” July 
31, 2018, https://www.whitehouse.gov/wp-content/uploads/2018/07/M-18-22.pdf. 


34 Dr. Matthijs Broer, Chief Technology Officer, Central Intelligence Agency, Comments at Defense One Summit, 
November 9, 2017. 


35 Testimony of Paul Scharre, House Armed Services Committee, Subcommittee on Emerging Threats and Capabilities, 
Hearing on China’s Pursuit of Emerging Technologies. 
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components. For example, in the FY2019 National Defense Authorization Act, AI funding is 
spread throughout the PEs for the High Performance Computing Modernization Program and 
Dominant Information Sciences and Methods, among others.*” On the other hand, a dedicated PE 
for AI may lead to a false precision, as it may be challenging to identify exact investments in 
enabling technologies like AI. 


The lack of an official U.S. government or DOD definition of AI could further complicate such an 
assessment. Indeed, DOD’s Office of Inspector General (DODIG) found that each of the military 
services uses a different definition of AI, and, in some cases, different individuals within the same 
military service were using different definitions of AI. This results in inconsistencies in the types 
of projects that are reported to—and overseen by—the JAIC.** DODIG additionally 
recommended that DOD establish “a process to accurately account for AI projects” and “a formal 
strategy for collaborating between the military services and DOD components on similar AI 
projects.”°? 


Congress may also consider specific policies for the development and use of military AI 
applications. Many experts fear that the pace of AI technology development is moving faster than 
the speed of policy implementation. Former Chairman of the House Armed Services Committee 
Representative Mac Thornberry has echoed this sentiment, stating, “It seems to me that we’re 
always a lot better at developing technologies than we are the policies on how to use them.’”“° 
Congress may assess the need for new policies or modifications to existing laws to account for AI 
developments and ensure that AI applications are free from bias.*! Perhaps the most immediate 
policy concern among AI analysts is the absence of an independent entity to develop and enforce 
AI safety standards and to oversee government-wide AI research. Former Secretary of Defense 
Ashton B. Carter, for example, has suggested the need for an “AI czar” to coordinate such 
efforts.*° 


Relatedly, Congress may consider debating policy options on the development and fielding of 
Lethal Autonomous Weapons Systems (LAWS), which may use AI to select and engage targets. 
Since 2014, the United States has participated in international discussions of LAWS at the United 
Nations (U.N.) Convention on Certain Conventional Weapons (CCW). Approximately 30 state 
parties have called for a treaty banning “fully autonomous weapon systems” due to ethical 
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considerations, while others have called for formal regulations or political declarations.“ Some 
analysts are concerned that efforts to ban or regulate LAWS could impose strict controls on AI 
applications that could be adapted for lethal use, thereby stifling development of other useful 
military—or even commercial—technology. During recent testimony to the U.N., one expert 
stated, “If we agree to foreswear some technology, we could end up giving up some uses of 
automation that could make war more humane. On the other hand a headlong rush into a future of 
increasing autonomy with no discussion of where it is taking us, is not in humanity’s interest 
either.” He suggested the leading question for considering military AI applications ought to be, 
“What role do we want humans to play in wartime decision making?”*» 


Congress may consider the growth of international competition in the AI market and the danger 
of foreign exploitation of U.S. AI technology for military purposes. In particular, the Chinese 
government is reported to be aggressively pursuing AI investments in the United States. Amid 
growing scrutiny of transactions involving Chinese firms in the semiconductor industry, in 
September 2017 President Trump, following the recommendation of the Committee on Foreign 
Investment in the United States (CFIUS), blocked a Chinese firm from acquiring Lattice 
Semiconductor, a U.S. company that manufactures chips that are a critical design element for AI 
technology.*° In this way, some experts believe that CFIUS may provide a means of protecting 
strategically significant technologies like AI.“ Indeed, the Foreign Investment Risk Review 
Modernization Act of 2018 (FIRRMA) expands CFIUS’s ability to review certain foreign 
investments, including those involving “emerging and foundational technologies.” It also 
authorized CFIUS to consider “whether a covered transaction involves a country of special 
concern that has a demonstrated or declared strategic goal of acquiring a type of critical 
technology or critical infrastructure that would affect United States leadership in areas related to 
national security.”“* Congress may monitor the implementation of FIRRMA and assess whether 
additional reforms might be necessary to maintain effective congressional oversight of sensitive 
transactions. 


In addition, many analysts believe that it may be necessary to reform federal data policies 
associated with AI. Large data pools serve as the training sets needed for building many AI 
systems, and government data may be particularly important in developing military AI 


44 See “Country Views on Killer Robots,” Campaign to Stop Killer Robots, April 13, 2018, 
https://www.stopkillerrobots.org/wp-content/uploads/2018/04/KRC_Country Views_13Apr2018.pdf; and U.N. CCW 
Working Papers and Statements at https://www.unog.ch/__80256ee600585943 nsf/(httpPages)/ 

7¢335e7 ldfcb29d1c1258243003e8724?OpenDocument&ExpandSection=3#_Section3. 


45 Paul Scharre, Remarks to the United Nations, Group of Governmental Experts on Lethal Autonomous Weapons 
Systems, November 15, 2017, Geneva, Switzerland, https://s3.amazonaws.com/files.cnas.org/documents/Scharre- 
Remarks-to-UN-on-Autonomous-Weapons-15-Nov-2017.pdf?mtime=2017 1120095806. For more information on 
LAWS, see CRS Report R44466, Lethal Autonomous Weapon Systems: Issues for Congress, by Nathan J. Lucas. 
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applications. However, some analysts have observed that much of this data is classified, access- 
controlled, or otherwise protected on privacy grounds. These analysts contend that Congress 
should implement a new data policy that balances data protection and privacy with the need to 
fuel AI development.” Similarly, DOD’s October 2020 data strategy seeks to improve the 
department’s management of data by making data more visible, accessible, understandable, 
linked, trustworthy, interoperable, and secure.~° 


Closely related, AI development may increase the imperative for strict security standards. As 
discussed later in this report, AI algorithms are vulnerable to bias, theft, and manipulation, 
particularly if the training data set is not adequately curated or protected. During a February 2018 
conference with defense industry CEOs, then-Deputy Defense Secretary Patrick Shanahan 
advocated for higher cybersecurity standards in the commercial sector, stating, “[W]e want the 
bar to be so high that it becomes a condition of doing business.”*! Some leading commercial 
technology companies have issued similar calls for increased scrutiny, with Microsoft president 
Brad Smith arguing that a lack of regulation in this area could lead to “a commercial race to the 
bottom, with tech companies forced to choose between social responsibility and market 


success.” 


Finally, commercial companies have long cited the potential loss of intellectual property rights as 
a key impediment to partnering with DOD. In recognition of this issue, Section 813 of the 
FY2016 NDAA established a “government-industry advisory panel” to provide recommendations 
on technical data rights and intellectual property reform.™ The panel’s report, released in 
November 2018, offers a number of recommendations, including increased training in intellectual 
property rights for acquisitions professionals and a pilot program for intellectual property 
valuation in the procurement process.™4 


AI Applications for Defense 


DOD is considering a number of diverse applications for AI. Currently, AI R&D is being left to 
the discretion of research organizations in the individual services, as well as to DARPA and the 
Intelligence Advanced Research Projects Agency (IARPA). However, DOD components are 
required to coordinate with the JAIC regarding any planned AI initiatives costing more than $15 
million annually.” In addition, the JAIC has been tasked with overseeing the National Mission 


49 Alexander Velez-Green and Paul Scharre, “The United States Can Be a World Leader in AI. Here’s How.,” The 
National Interest, November 2, 2017, https://nationalinterest.org/feature/the-united-states-can-be-world-leader-ai- 
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145779/?oref=d-river. For more on cybersecurity legislation, see CRS Report R42114, Federal Laws Relating to 
Cybersecurity: Overview of Major Issues, Current Laws, and Proposed Legislation, by Eric A. Fischer. 
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Initiatives (NMJ), projects that leverage AI to address pressing operational challenges.°° The 
JAIC began work on its first two NMIs—predictive maintenance and humanitarian assistance and 
disaster relief—in 2019. According to former JAIC acting director Nand Mulchandani, the JAIC 
then shifted its focus to joint warfighting and, in May 2020, awarded an $800 million contract to 
Booz Allen Hamilton to “bring AI to the battlefield.”°’ Current JAIC director Lt. Gen. Michael 
Groen has stated that the JAIC is now focused on further developing databases and software tools 
for use in AI projects through the department.** AI is being incorporated into a number of other 
intelligence, surveillance, and reconnaissance applications, as well as in logistics, cyberspace 
operations, information operations, command and control, semiautonomous and autonomous 
vehicles, and lethal autonomous weapon systems. 


Intelligence, Surveillance, and Reconnaissance 


Al is expected to be particularly useful in intelligence due to the large data sets available for 
analysis.°’ For example, Project Maven is intended to incorporate computer vision and AI 
algorithms into intelligence collection cells that would comb through footage from uninhabited 
aerial vehicles and automatically identify hostile activity for targeting. In this capacity, AI is 
intended to automate the work of human analysts who currently spend hours sifting through drone 
footage for actionable information, potentially freeing analysts to make more efficient and timely 
decisions based on the data. 


The intelligence community also has a number of publicly acknowledged AI research projects in 
progress. The Central Intelligence Agency alone has around 140 projects in development that 
leverage AI in some capacity to accomplish tasks such as image recognition and predictive 
analytics.°' IARPA is sponsoring several AI research projects intended to produce other analytic 
tools within the next four to five years. Some examples include developing algorithms for 
multilingual speech recognition and translation in noisy environments, geo-locating images 
without the associated metadata, fusing 2-D images to create 3-D models, and building tools to 
infer a building’s function based on pattern-of-life analysis. 


establishment_of_the_joint_artificial_intelligence_center_osd008412-18_r.... pdf. 
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Logistics 


AI may have future utility in the field of military logistics. The Air Force, for example, is 
beginning to use AI for predictive aircraft maintenance. Instead of making repairs when an 
aircraft breaks or in accordance with standardized fleet-wide maintenance schedules, the Air 
Force is testing an Al-enabled approach that tailors maintenance schedules to the needs of 
individual aircraft. This approach, currently used by the F-35’s Autonomic Logistics Information 
System, extracts real-time sensor data embedded in the aircraft’s engines and other onboard 
systems and feeds the data into a predictive algorithm to determine when technicians need to 
inspect the aircraft or replace parts. 


Similarly, the Army’s Logistics Support Activity (LOGSA) has contracted IBM’s Watson (the 
same AI software that defeated two Jeopardy champions) to develop tailored maintenance 
schedules for the Stryker fleet based on information pulled from the 17 sensors installed on each 
vehicle. In September 2017, LOGSA began a second project that will use Watson to analyze 
shipping flows for repair parts distribution, attempting to determine the most time- and cost- 
efficient means to deliver supplies. This task is currently done by human analysts, who have 
saved the Army around $100 million a year by analyzing just 10% of shipping requests; with 
Watson, the Army will have the ability to analyze 100% of shipping requests, potentially 
generating even greater cost savings in a shorter period of time.™ 


Cyberspace Operations 


Al is likely to be a key technology in advancing military cyber operations. In his 2016 testimony 
before the Senate Armed Services Committee, Commander of U.S. Cyber Command Admiral 
Michael Rogers stated that relying on human intelligence alone in cyberspace is “a losing 
strategy.” He later clarified this point, stating, “If you can’t get some level of AI or machine 
learning with the volume of activity you’re trying to understand when you’re defending networks 
... you are always behind the power curve.” Conventional cybersecurity tools look for historical 
matches to known malicious code, so hackers only have to modify small portions of that code to 
circumvent the defense. Al-enabled tools, on the other hand, can be trained to detect anomalies in 
broader patterns of network activity, thus presenting a more comprehensive and dynamic barrier 
to attack.” 


DARPA’s 2016 Cyber Grand Challenge demonstrated the potential power of Al-enabled cyber 
tools. The competition challenged participants to develop AI algorithms that could autonomously 
“detect, evaluate, and patch software vulnerabilities before [competing teams] have a chance to 
exploit them’”—all within a matter of seconds, rather than the usual months.® The challenge 
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demonstrated not only the potential speed of Al-enabled cyber tools but also the potential ability 
of a singular algorithm to play offense and defense simultaneously. These capabilities could 
provide a distinct advantage in future cyber operations. 


Information Operations and “Deep Fakes” 


Al is enabling increasingly realistic photo, audio, and video forgeries, or “deep fakes,” that 
adversaries could deploy as part of their information operations. Indeed, deep fake technology 
could be used against the United States and U.S. allies to generate false news reports, influence 
public discourse, erode public trust, and attempt to blackmail diplomats.” Although most 
previous deep fakes have been detectable by experts, the sophistication of the technology is 
progressing to the point that it may soon be capable of fooling forensic analysis tools.”! 


In order to combat deep fake technologies, DARPA has launched the Media Forensics (MediFor) 
project, which seeks to “automatically detect manipulations, provide detailed information about 
how these manipulations were performed, and reason about the overall integrity of visual 
media.””? MediFor has developed some initial tools for identifying Al-produced forgeries, but as 
one analyst has noted, “a key problem ... is that machine-learning systems can be trained to 
outmaneuver forensics tools.”” For this reason, DARPA plans to host follow-on contests to 
ensure that forensic tools keep pace with deep fake technologies.” It has also launched SemaFor, 
which seeks to develop algorithms that will automatically detect, attribute, and characterize (1.e., 
identify as either benign or malicious) various types of deep fakes. 


Artificial intelligence could also be used to create full “digital patterns-of-life,” in which an 
individual’s digital “footprint” is “merged and matched with purchase histories, credit reports, 
professional resumes, and subscriptions” to create a comprehensive behavioral profile of 
servicemembers, suspected intelligence officers, government officials, or private citizens.” As in 
the case of deep fakes, this information could, in turn, be used for targeted influence operations or 
blackmail. 


Command and Control 


The U.S. military is seeking to exploit AI’s analytic potential in the area of command and control. 
DOD is developing various systems in support of its concept of Joint All Domain Command and 
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Control (JADC2), which aims to centralize planning and execution of air-, space-, cyberspace-, 
sea-, and land-based operations.’° In the immediate future, AI may be used to fuse data from 
sensors in all of these domains to create a single source of information, also known as a “common 
operating picture,” for decisionmakers.’’ Currently, information available to decisionmakers 
comes in diverse formats from multiple platforms, often with redundancies or unresolved 
discrepancies. An AI-enabled common operating picture would theoretically combine this 
information into one display, providing a comprehensive picture of friendly and enemy forces, 
and automatically resolving variances from input data. Such a system could eventually enable 
“any sensor to provide data to any shooter from any service, ally, or partner ... to achieve effects 
against any target.”’* The services have a number of related programs that are designed to 
demonstrate the capabilities needed to execute JADC2, including the Army’s Project 
Convergence and the Air Force’s Advanced Battle Management System.” Similarly, DARPA’s 
Mosaic Warfare program seeks to leverage AI to network systems and sensors, prioritize 
incoming sensor data, and autonomously determine the optimal composition of forces. °° 


Future AI systems may be used to identify communications links cut by an adversary and find 
alternative means of distributing information. As the complexity of AI systems matures, AI 
algorithms may also be capable of providing commanders with a menu of viable courses of action 
based on real-time analysis of the battle-space, potentially improving the quality and speed of 
wartime decisionmaking.*! 


Semiautonomous and Autonomous Vehicles 


All U.S. military services are working to incorporate AI into semiautonomous and autonomous 
vehicles, including fighter aircraft, drones, ground vehicles, and naval vessels. AI applications in 
this field are similar to those for commercial semiautonomous vehicles, which use AI 
technologies to perceive the environment, recognize obstacles, fuse sensor data, plan navigation, 
and even communicate with other vehicles.* 


The Air Force Research Lab completed phase-two tests of its Loyal Wingman program, which 
pairs an older-generation, uninhabited fighter jet (in this case, an F-16) with an inhabited F-35 or 
F-22. During this event, the uninhabited F-16 test platform autonomously reacted to events that 
were not preprogrammed, such as weather and unforeseen obstacles.’ As the program progresses, 
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AI may enable the “loyal wingman” to accomplish tasks for its inhabited flight lead, such as 
jamming electronic threats or carrying extra weapons.** 


The Army and the Marine Corps tested prototypes of similar vehicles that follow soldiers or 
vehicles around the battlefield to accomplish independent tasks.*° For example, the Marine Corps’ 
Multi-Utility Tactical Transport (MUTT) is a remote-controlled, ATV-sized vehicle capable of 
carrying hundreds of pounds of extra equipment. Although the system is not autonomous in its 
current configuration, the Marine Corps intends for follow-on systems to have greater 
independence.* Likewise, the Army plans to field a number of Robotic Combat Vehicles (RCVs) 
with different types of autonomous functionality, including navigation, surveillance, and IED 
removal. These systems are to be deployed as “wingmen” for the Optionally Manned Fighting 
Vehicle.*’ 


DARPA completed testing of the Anti-Submarine Warfare Continuous Trail Unmanned Vessel 
prototype, or “Sea Hunter,” in early 2018 before transitioning program development to the 
Navy. Sea Hunter has since been integrated into Surface Development Squadron 1, which has 
been tasked with overseeing “fleet familiarization, training and tactics development of [unmanned 
surface vessels].” If Sea Hunter enters into service, it would provide the Navy with the ability to 
autonomously navigate the open seas, swap out modular payloads, and coordinate missions with 
other unmanned vessels—all while providing continuous submarine-hunting coverage for months 
at a time.® Some analysts estimate that Sea Hunter would cost around $20,000 a day to operate, 
in contrast to around $700,000 for a traditionally inhabited destroyer.” In addition, the Navy has 
launched the Rapid Autonomy Integration Lab (RAIL) “to develop, test, certify and deploy new 
and updated autonomous capabilities.””! 


DOD is testing other AI-fueled capabilities to enable cooperative behavior, or swarming. 
Swarming is a unique subset of autonomous vehicle development, with concepts ranging from 
large formations of low-cost vehicles designed to overwhelm defensive systems to small 
squadrons of vehicles that collaborate to provide electronic attack, fire support, and localized 
navigation and communication nets for ground-troop formations.” A number of different swarm 
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capabilities are currently under development. For example, in November 2016, the Navy 
completed a test of an Al-enabled swarm of five unmanned boats that cooperatively patrolled a 4- 
by-4-mile section of the Chesapeake Bay and intercepted an “intruder” vessel. The results of this 
experiment may lead to AI technology adapted for defending harbors, hunting submarines, or 
scouting in front of a formation of larger ships.” The Navy also plans to test swarms of 
underwater drones, and the Strategic Capabilities Office has successfully tested a swarm of 103 
air-dropped micro-drones.™ 


Swarm Characteristics’ 
Autonomous (not under centralized control) 


Capable of sensing their local environment and other 
nearby swarm participants 


Able to communicate locally with others in the swarm 


Able to cooperate to perform a given task 


Lethal Autonomous Weapon Systems (LAWS) 


Lethal Autonomous Weapon Systems (LAWS) are a special class of weapon systems that use 
sensor suites and computer algorithms to independently identify a target and employ an onboard 
weapon system to engage and destroy the target without manual human control of the system. 
Although these systems are not yet in widespread development, they could enable military 
operations in communications-degraded or -denied environments in which traditional systems 
may not be able to operate. The U.S. military does not currently have LAWS in its inventory, 
although there are no legal prohibitions on the development of LAWS. 


DOD Directive 3000.09, “Autonomy in Weapon Systems,” outlines department policies for 
semiautonomous and autonomous weapon systems.” The directive requires that all systems, 
regardless of classification, be designed to “allow commanders and operators to exercise 
appropriate levels of human judgment over the use of force” and to successfully complete the 
department’s weapons review process.” Any changes to the system’s operating state require that 
the system go through the weapons review process again to ensure that it has retained the ability 
to operate as intended. Autonomous weapons and a limited type of semiautonomous weapons 
must additionally be approved before both development and fielding by the Under Secretary of 
Defense for Policy, the Chairman of the Joint Chiefs of Staff, and either the Under Secretary of 
Defense for Acquisition and Sustainment or the Under Secretary of Defense for Research and 
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Engineering. Human-supervised autonomous weapons used for point defense of manned 
installations or platforms—but that do not target humans—and autonomous weapons that “apply 
non-lethal, non-kinetic force, such as some forms of electronic attack, against materiel targets” 
are exempted from this senior-level review.”® 


Despite this policy, some senior military and defense leaders have expressed concerns about the 
prospect of fielding LAWS. For example, in 2017 testimony before the Senate Armed Services 
Committee, then-Vice Chairman of the Joint Chiefs of Staff General Paul Selva stated, “I do not 
think it is reasonable for us to put robots in charge of whether or not we take a human life.””? 
Regardless, General Selva explained that the military will be compelled to address the 
development of this class of technology in order to find its vulnerabilities, given the fact that 
potential U.S. adversaries are pursuing LAWS.’ Indeed, as former Secretary of Defense Mark 
Esper has noted, “Chinese weapons manufacturers are selling drones advertised as capable of full 
autonomy, including the ability to conduct lethal targeted strikes.”'”! 


Military AI Integration Challenges 


From the Cold War era until recently, most major defense-related technologies, including nuclear 
technology, the Global Positioning System (GPS), and the internet, were first developed by 
government-directed programs before later spreading to the commercial sector." Indeed, 
DARPA’s Strategic Computing Initiative invested over $1 billion between 1983 and 1993 to 
develop the field of artificial intelligence for military applications, but the initiative was 
ultimately cancelled due to slower-than-anticipated progress.! Today, commercial companies— 
sometimes building on past government-funded research—are leading AI development, with 
DOD later adapting their tools for military applications.'“ Noting this dynamic, one AI expert 
commented, “It is unusual to have a technology that is so strategically important being developed 
commercially by a relatively small number of companies.”! In addition to the shift in funding 
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sources, a number of challenges related to technology, process, personnel, and culture continue to 
impede the adoption of AI for military purposes. 


Technology 


A wide variance exists in the ease of adaptability of commercial AI technology for military 
purposes. In some cases, the transition is relatively seamless. For example, the aforementioned 
aircraft maintenance algorithms, many of which were initially developed by the commercial 
sector, will likely require only minor data adjustments to account for differences between aircraft 
types. In other circumstances, significant adjustments are required due to the differences between 
the structured civilian environments for which the technology was initially developed and more 
complex combat environments. For example, commercial semiautonomous vehicles have largely 
been developed in and for data-rich environments with reliable GPS positions, comprehensive 
terrain mapping, and up-to-date information on traffic and weather conditions obtained from 
other networked vehicles.!° In contrast, the military variant of such a vehicle would need to be 
able to operate in locations where map data are comparatively poor and in which GPS positioning 
may be inoperable due to adversary jamming. Moreover, semiautonomous or autonomous 
military ground vehicles would likely need the ability to navigate off-road in rough terrain—a 
capability not inherent in most commercial vehicles.!” 


Process 


Standing DOD processes—including those related to standards of safety and performance, 
acquisitions, and intellectual property and data rights—present another challenge to the 
integration of military AI. Often, civilian and military standards of safety and performance are 
either not aligned or are not easily transferable. A failure rate deemed acceptable for a civilian AI 
application may be well outside of tolerances in a combat environment—or vice versa. In 
addition, a recent research study concluded that unpredictable AI failure modes will be 
exacerbated in complex environments, such as those found in combat.!°* Collectively, these 
factors may create another barrier for the smooth transfer of commercially developed AI 
technology to DOD. 


DOD may need to continue to adjust its acquisitions process to account for rapidly evolving 
technologies such as AI.'° A 2017 internal study of the process found that it takes an average of 
91 months to move from the initial Analysis of Alternatives, defining the requirements for a 
system, to an Initial Operational Capability.'!° In contrast, commercial companies typically 
execute an iterative development process for software systems like AI, delivering a product in six 
to nine months.''! These findings prompted DOD to issue an interim software acquisition policy 
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intended to “[simplify] the acquisition model to enable continuous integration and delivery of 
software capability on timelines relevant to the Warfighter/end user.”!!” 


Furthermore, the commercial companies that are often at the forefront of AI innovation may be 
reluctant to partner with DOD due to the complexity of the defense acquisition process. A 
Government Accountability Office (GAO) study of this issue found that, of 12 U.S. commercial 
companies who choose not to do business with DOD, all 12 cited the complexity of the defense 
acquisition process as a rationale for their decision.!'? DOD has created a number of avenues for 
“rapid-acquisitions”—including the Strategic Capabilities Office, the Defense Innovation Unit, 
and Project Maven—that are intended streamline cumbersome processes and accelerate the 
acquisitions timeline.!!* Project Maven, for example, was established in April 2017; by 
December, the team was fielding a commercially acquired prototype AI system in combat.'"° 
Although some analysts argue that these are promising developments, critics point out that the 
department must replicate the results achieved by Project Maven at scale and implement more 
comprehensive acquisitions reform. !!6 


Commercial technology companies are also often reluctant to partner with DOD due to concerns 
about intellectual property and data rights.” As an official interviewed for a 2017 GAO report on 
broader challenges in military acquisitions noted, intellectual property is the “life blood” of 
commercial technology companies, yet “DOD is putting increased pressure on companies to grant 
unlimited technical data and software rights or government purpose rights rather than limited or 
restricted rights.”''® In an effort to manage these concerns, DOD released an instruction that 
“establishes policy, assigns responsibilities, and prescribes procedures for the acquisition, 
licensing, and management of IP.”'!? The instruction additionally establishes a DOD IP Cadre to 
advise and assist the acquisition workforce on matters related to IP and calls for the development 
of an IP strategy to “identify and manage the full spectrum of IP and related matters” for each 
acquisition program. '° 
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Personnel 


Some reports indicate that DOD and the defense industry also face challenges when it comes to 
recruiting and retaining personnel with expertise in AI due to research funding and salaries that 
significantly lag behind those of commercial companies.'”! Other reports suggest that such 
challenges stem from quality-of-life factors, as well as from a belief among many technology 
workers that “they can achieve large-scale change faster and better outside the government than 
within it.”!?? This sentiment echoes the findings of the National Security Commission on 
Artificial Intelligence, which notes that “AI experts would be willing to serve in government if 
officials could create a more compelling sense of purpose and a technical environment within 
government that would maximize their talents.”!*? Regardless, observers note that if DOD and 
defense industry are unable to recruit and retain the appropriate experts, military AI applications 
could be delayed, “deficient, or lacking in appropriate safeguards and testing.” 1 


To address these challenges, the Obama Administration launched the Defense Digital Service in 
2015 as a means of recruiting private sector technology workers to serve in DOD for one to two 
year assignments—a “tour of duty for nerds,” according to former director Chris Lynch.'7 
Similarly, former Deputy Secretary of Defense Bob Work has proposed an “AI Training Corps,” 
in which DOD “would pay for advanced technical education in exchange for two days a month of 
training with government systems and two weeks a year for major exercises.” Participants in the 
program could additionally be called to government service in the event of a national 

emergency. '”° Other analysts have recommended the establishment of new military training and 
occupational specialties to cultivate AI talent, as well as the creation of government fellowships 
and accelerated promotion tracks to reward the most talented technology workers.!’ 


Culture 


An apparent cultural divide between DOD and commercial technology companies may also 
present challenges for AI adoption. A recent survey of leadership in several top Silicon Valley 
companies found that nearly 80% of participants rated the commercial technology community’s 
relationship with DOD as poor or very poor.'** This was due to a number of factors, including 
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process challenges, perceptions of mutual distrust, and differences between DOD and commercial 
incentive structures. !”° 


Moreover, some companies are refusing to work with DOD due to ethical concerns over the 
government’s use of AI in surveillance or weapon systems. Notably, Google canceled existing 
government contracts for two robotics companies it acquired—Boston Dynamics and Schaft— 
and prohibited future government work for DeepMind, a Google-acquired AI software startup.!*° 
In May 2018, Google employees successfully lobbied the company to withdraw from Project 
Maven and refrain from further collaboration with DOD.'*! Other companies, however, have 
pledged to continue supporting DOD contracts, with Amazon CEO Jeff Bezos noting that “if big 
tech companies are going to turn their back on the U.S. Department of Defense, this country is 
going to be in trouble.”!? 


Cultural factors within the defense establishment itself may also impede AI integration. The 
integration of AI into existing systems alters standardized procedures and upends well-defined 
personnel roles. Members of Project Maven have reported a resistance to AI integration because 
integration can be disruptive without always providing an immediately recognizable benefit.!** 
Deputy Director for CIA technology development Dawn Meyerriecks has also expressed concern 
about the willingness of senior leaders to accept Al-generated analysis, arguing that the defense 
establishment’s risk-averse culture may pose greater challenges to future competitiveness than the 
pace of adversary technology development.'™* 


Finally, some analysts are concerned that DOD will not capitalize on AI’s potential to produce 
game-changing warfighting benefits and will instead simply use AI to incrementally improve 
existing processes or reinforce current operational concepts. Furthermore, the services may reject 
certain AI applications altogether if the technology threatens service-favored hardware or 
missions." Members of Congress may explore the complex interaction of these factors as DOD 
moves beyond the initial stages of AI adoption. 


International Competitors 


As military applications for AI grow in scale and complexity, many in Congress and the defense 
community are becoming increasingly concerned about international competition. In his opening 
comments at “The Dawn of AT” hearing before the Senate Subcommittee on Space, Science, and 
Competitiveness, Senator Ted Cruz stated, “Ceding leadership in developing artificial intelligence 
to China, Russia, and other foreign governments will not only place the United States at a 
technological disadvantage, but it could have grave implications for national security.”’'*° 
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Since at least 2016, AI has been consistently identified as an “emerging and disruptive 
technology” at the Senate Select Intelligence Committee’s annual hearing on the “Worldwide 
Threat Assessment.”!%” In his written testimony for the 2017 hearing, Director of National 
Intelligence Daniel Coates asserted, “The implications of our adversaries’ abilities to use AI are 
potentially profound and broad. They include an increased vulnerability to cyberattack, difficulty 
in ascertaining attribution, facilitation of advances in foreign weapon and intelligence systems, 
the risk of accidents and related liability issues, and unemployment.”'*® Consequently, it may be 
important for Congress to understand the state of rival AI development—particularly because 
U.S. competitors may have fewer moral, legal, or ethical qualms about developing military AI 
applications.!* 


China 


China is by far the United States’ closest competitor in the international AI market.'#° China’s 
2017 “Next Generation AI Development Plan” describes AI as a “strategic technology” that has 
become a “focus of international competition.”'*' According to the document, China will seek to 
develop a core AI industry worth over 150 billion RMB!#—or approximately $21.7 billion—by 
2020 and will “firmly seize the strategic initiative” and reach “world leading levels” of AI 
investment by 2030. 


Recent Chinese achievements in the field demonstrate China’s potential to realize its goals for AI 
development. In 2015, China’s leading AI company, Baidu, created AI software capable of 
surpassing human levels of language recognition, almost a year in advance of Microsoft, the 
nearest U.S. competitor.'*? In 2016 and 2017, Chinese teams won the top prize at the Large Scale 
Visual Recognition Challenge, an international competition for computer vision systems.'*+ Many 
of these systems are now being integrated into China’s domestic surveillance network and social 
credit system, which aims to monitor and, based on social behavior, “grade” every Chinese 
citizen by 2021.1“ 
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China is researching various types of air, land, sea, and undersea autonomous vehicles. In the 
spring of 2017, a civilian Chinese university with ties to the military demonstrated an Al-enabled 
swarm of 1,000 uninhabited aerial vehicles at an airshow. A media report released after the fact 
showed a computer simulation of a similar swarm formation finding and destroying a missile 
launcher.'*° Open-source publications indicate that the Chinese are also developing a suite of AI 
tools for cyber operations.'*” 


Chinese development of military AI is influenced in large part by China’s observation of U.S. 
plans for defense innovation and fears of a widening “generational gap” in comparison to the U.S. 
military. '** Similar to U.S. military concepts, the Chinese aim to use AI for exploiting large troves 
of intelligence, generating a common operating picture, and accelerating battlefield 
decisionmaking.'*” The close parallels between U.S. and Chinese AI development have some 
DOD leaders concerned about the prospects for retaining conventional U.S. military superiority 
as envisioned in current defense innovation guidance.'*° 


Analysts do, however, point to a number of differences that may influence the success of military 
AI adoption in China. Significantly, unlike the United States, China has not been involved in 
active combat for several decades. While on the surface this may seem like a weakness, some 
argue that it may be an advantage, enabling the Chinese to develop more innovative concepts of 
operation. On the other hand, Chinese military culture, which is dominated by centralized 
command authority and mistrust of subordinates, may prove resistant to the adoption of 
autonomous systems or the integration of Al-generated decisionmaking tools.'*! 


China’s management of its AI ecosystem stands in stark contrast to that of the United States.!°? In 
general, few boundaries exist between Chinese commercial companies, university research 
laboratories, the military, and the central government. As a result, the Chinese government has a 
direct means of guiding AI development priorities and accessing technology that was ostensibly 
developed for civilian purposes. To further strengthen these ties, the Chinese government created 
a Military-Civil Fusion Development Commission in 2017, which is intended to speed the 
transfer of AI technology from commercial companies and research institutions to the military. 19 
In addition, the Chinese government is leveraging both lower barriers to data collection and lower 
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costs to data labeling to create the large databases on which AI systems train.'™ According to one 
estimate, China is on track to possess 30% of the world’s share of data by 2030.!°° 


China’s centrally directed effort is fueling speculation in the U.S. AI market, where China is 
investing in companies working on militarily relevant AI applications—potentially granting it 
lawful access to U.S. technology and intellectual property.'* Figure 2 depicts Chinese venture 
capital investment in U.S. AI companies between 2010 and 2017, totaling an estimated $1.3 
billion. The CFIUS reforms introduced in FIRRMA are intended to provide increased oversight of 
such investments to ensure that they do not threaten national security or grant U.S. competitors 
undue access to critical technologies.'°’ Indeed, early analysis suggests a correlation between 
CFIUS reforms and a reduction in Chinese investment in U.S. AI companies.'* 


Figure 2. Chinese Investment in U.S.Al Companies, 2010-2017 
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Source: Michael Brown and Pavneet Singh, China’s Technology Transfer Strategy: How Chinese Investments in 
Emerging Technology Enable a Strategic Competitor to Access the Crown Jewels of U.S. Innovation, Defense 
Innovation Unit Experimental, January 2018, p. 29, 
https://admin.govexec.com/media/diux_chinatechnologytransferstudy_jan_2018_(1).pdf. 


Even with these reforms, however, China may likely gain access to U.S. commercial 
developments in AI given its extensive history of industrial espionage and cyber theft. China has 
reportedly stolen design plans in the past for a number of advanced military technologies and 
continues to do so despite the 2015 U.S.-China Cyber Agreement, in which both sides agreed that 
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“neither country’s government will conduct or knowingly support cyber-enabled theft of 
intellectual property.”!® Furthermore, as Assistant Attorney General John Demers has testified to 
Congress, “from 2011-2018, more than 90 percent of the [Justice] Department’s cases alleging 
economic espionage by or to benefit a state involve China, and more than two-thirds of the 
Department’s theft of trade secrets cases have had a nexus to China.” 16 


While most analysts view China’s unified, whole-of-government effort to develop AI as having a 
distinct advantage over the United States’ AI efforts, many contend that it does have 
shortcomings. For example, some analysts characterize the Chinese government’s funding 
management as inefficient. They point out that the system is often corrupt, with favored research 
institutions receiving a disproportionate share of government funding, and that the government 
has a potential to overinvest in projects that produce surpluses that exceed market demand. !®! 


In addition, China faces challenges in recruiting and retaining AI engineers and researchers. Over 
half of the data scientists in the United States have been working in the field for over 10 years, 
while roughly the same proportion of data scientists in China have less than 5 years of 
experience. Furthermore, fewer than 30 Chinese universities produce Al-focused experts and 
research products. Although China surpassed the United States in the quantity of research 
papers produced from 2011 to 2015, the quality of its published papers, as judged by peer 
citations, ranked 34" globally.'® China is, however, making efforts to address these deficiencies, 
with a particular focus on the development of military AI applications. Indeed, the Beijing 
Institute of Technology—one of China’s premier institutes for weapons research—recently 
established the first educational program in military AI in the world.!™ 


Some experts believe that China’s intent to be the first to develop military AI applications may 
result in comparatively less safe applications, as China will likely be more risk-acceptant 
throughout the development process. These experts stated that it would be unethical for the U.S. 
military to sacrifice safety standards for the sake of external time pressures, but that the United 
States’ more conservative approach to AI development may result in more capable systems in the 
long run.'® 
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Russia 


Like China, Russia is actively pursuing military AI applications. At present, Russian AI 
development lags significantly behind that of the United States and China, with no Russian AI 
startup ranking in the top 100.'°° However, Russia is initiating plans to close the gap. As part of 
this effort, Russia has released a national strategy for artificial intelligence, which outlines 5- and 
10-year benchmarks for improving the country’s AI expertise, educational programs, datasets, 
infrastructure, and legal regulatory system.'®’ Russia will also continue to pursue its 2008 defense 
modernization agenda, with the aim of robotizing 30% of its military equipment by 2025.168 


Russia is establishing a number of organizations devoted to the development of military AI. In 
March 2018, the Russian government released a 10-point AI agenda, which calls for the 
establishment of an AI and Big Data consortium, a Fund for Analytical Algorithms and Programs, 
a state-backed AI training and education program, a dedicated AI lab, and a National Center for 
Artificial Intelligence, among other initiatives.’ In addition, Russia recently created a defense 
research organization, roughly equivalent to DARPA, dedicated to autonomy and robotics called 
the Foundation for Advanced Studies, and initiated an annual conference on “Robotization of the 
Armed Forces of the Russian Federation.”!’” Some analysts have noted that this recent 
proliferation of research institutions devoted to AI may, however, result in overlapping 


responsibilities and bureaucratic inertia, hindering AI development rather than accelerating it.'”! 


The Russian military has been researching a number of AI applications, with a heavy emphasis on 
semiautonomous and autonomous vehicles. In an official statement on November 1, 2017, Viktor 
Bondarev, chairman of the Federation Council’s Defense and Security Committee, stated that 
“artificial intelligence will be able to replace a soldier on the battlefield and a pilot in an aircraft 
cockpit” and later noted that “the day is nearing when vehicles will get artificial intelligence.” "7? 
Bondarev made these remarks in close proximity to the successful test of Nerehta, an uninhabited 
Russian ground vehicle that reportedly “outperformed existing [inhabited] combat vehicles.” 
Russia plans to use Nerehta as a research and development platform for AI and may one day 
deploy the system in combat, intelligence gathering, or logistics roles.'”* Russia has also 
reportedly built a combat module for uninhabited ground vehicles that is capable of autonomous 
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target identification—and, potentially, target engagement—and plans to develop a suite of AI- 
enabled autonomous systems.!”4 


In addition, the Russian military plans to incorporate AI into uninhabited aerial, naval, and 
undersea vehicles and is currently developing swarming capabilities.'” It is also exploring 
innovative uses of AI for remoting sensing and electronic warfare, including adaptive frequency 
hopping, waveforms, and countermeasures.!”° Finally, Russia has made extensive use of AI 
technologies for domestic propaganda and surveillance, as well as for information operations 
directed against the United States and U.S. allies, and can be expected to continue to do so in the 
future.'”” 


Despite its aspirations, analysts argue that it may be difficult for Russia to make significant 
progress in AI development given fluctuations in Russian military spending. In 2017, Russian 
military spending dropped by 20% in constant dollars, with subsequent cuts in 2018.178 Spending 
then rose in 2019, but is expected to drop in constant dollars from 2020 through 2022.'” In 
addition, many analysts note that Russian academics have produced few research papers on AI 
and that the Russian technology industry has yet to produce AI applications that are on par with 
those produced by the United States and China.'*° Others analysts counter that such factors may 
be irrelevant, arguing that while Russia has never been a leader in internet technology, it has still 
managed to become a notably disruptive force in cyberspace.!*! Russia may also be able to draw 
upon its growing technological cooperation with China. '*” 
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International Institutions 


A number of international institutions have examined issues surrounding AI, including the Group 
of Seven (G7), the Asia-Pacific Economic Cooperation (APEC), and the Organisation for 
Economic Co-operation and Development (OECD), which has developed principles for the 
development of trustworthy AI.!8 The U.N. CCW, however, has made the most concerted effort 
to consider certain military applications of AI, with a particular focus on LAWS. In general, the 
CCW is charged with “banning or restricting the use of specific types of weapons that are 
considered to cause unnecessary or unjustifiable suffering to combatants or to affect civilian 
populations” and has previously debated weapons such as mines, cluster munitions, and blinding 
lasers.'** The CCW began discussions on LAWS in 2014 with informal annual “Meetings of 
Experts.”!* In parallel, the International Committee of the Red Cross (ICRC) held similar 
gatherings of interdisciplinary experts on LAWS that produced reports for the CCW on technical, 
legal, moral, and humanitarian issues.'*° During the CCW’s April 2016 meeting, state parties 
agreed to establish a formal Group of Governmental Experts (GGE), with an official mandate to 
“assess questions related to emerging technologies in the area of LAWS.”!%’ Although the GGE 
has now convened five times, it has not produced an official definition of LAWS or issued official 
guidance for their development or use. As a result, one observer cautioned that the international 
community is in danger of “the pace of diplomacy falling behind the speed of technological 
advancement.”!** The U.S. government, however, maintains that existing international 
humanitarian law is sufficient to govern the development or use of LAWS.!*? 


AI Opportunities and Challenges 


AI poses a number of unique opportunities and challenges within a national security context. 
However, its ultimate impact will likely be determined by the extent to which developers, with 
the assistance of policymakers, are able to maximize its strengths while identifying options to 
limit its vulnerabilities. 
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Autonomy 


Many autonomous systems incorporate AI in some form. Such systems were a central focus of the 
Obama Administration’s “Third Offset Strategy,” a framework for preserving the U.S. military’s 
technological edge against global competitors.!”° Depending on the task, autonomous systems are 
capable of augmenting or replacing humans, freeing them up for more complex and cognitively 
demanding work. In general, experts assert that the military stands to gain significant benefits 
from autonomous systems by replacing humans in tasks that are “dull, dangerous, or dirty.”!?! 
Specific examples of autonomy in military systems include systems that conduct long-duration 
intelligence collection and analysis, clean up environments contaminated by chemical weapons, 
or sweep routes for improvised explosive devices.'” In these roles, autonomous systems may 
reduce risk to warfighters and cut costs, providing a range of value to DOD missions, as 
illustrated in Figure 3.'°? Some analysts argue these advantages create a “tactical and strategic 
necessity” as well as a “moral obligation” to develop autonomous systems.!™* 


Figure 3.Value of Autonomy to DOD Missions 
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Speed and Endurance 


Al introduces a unique means of operating in combat at the extremes of the time scale. It provides 
systems with an ability to react at gigahertz speed, which in turn holds the potential to 
dramatically accelerate the overall pace of combat.'” As discussed below, some analysts contend 
that a drastic increase in the pace of combat could be destabilizing—particularly if it exceeds 
human ability to understand and control events—and could increase a system’s destructive 
potential in the event of a loss of system control.!*° Despite this risk, some argue that speed will 
confer a definitive warfighting advantage, in turn creating pressures for widespread adoption of 
military AI applications.'”’ In addition, AI systems may provide benefits in long-duration tasks 
that exceed human endurance. For example, AI systems may enable intelligence gathering across 
large areas over long periods of time, as well as the ability to autonomously detect anomalies and 
categorize behavior." 


Scaling 


AI has the potential to provide a force-multiplying effect by enhancing human capabilities and 
infusing less expensive military systems with increased capability. For example, although an 
individual low-cost drone may be powerless against a high-tech system like the F-35 stealth 
fighter, a swarm of such drones could potentially overwhelm high-tech systems, generating 
significant cost-savings and potentially rendering some current platforms obsolete.!*? AI systems 
could also increase the productivity of individual servicemembers as the systems take over 
routine tasks or enable tactics like swarming that require minimal human involvement.” 


Finally, some analysts caution that the proliferation of AI systems may decouple military power 
from population size and economic strength. This decoupling may enable smaller countries and 
nonstate actors to have a disproportionately large impact on the battlefield if they are able to 
capitalize on the scaling effects of AI.?”! 


Information Superiority 


AI may offer a means to cope with an exponential increase in the amount of data available for 
analysis. According to one DOD source, the military operates over 11,000 drones, with each one 
recording “more than three NFL seasons worth” of high-definition footage each day.””? However, 
the department does not have sufficient people or an adequate system to comb through the data in 
order to derive actionable intelligence analysis. 
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This issue will likely be exacerbated in the future as data continue to accumulate. According to 
one study, in 2020, every human on the planet will generate 1.7 megabytes of information every 
second, growing the global pool of data to almost 44.0 zettabytes, up from 4.4 zettabytes in 
2015.73 Al-powered intelligence systems may provide the ability to integrate and sort through 
large troves of data from different sources and geographic locations to identify patterns and 
highlight useful information, significantly improving intelligence analysis.” In addition, AI 
algorithms may generate their own data to feed further analysis, accomplishing tasks like 
converting unstructured information from polls, financial data, and election results into written 
reports. AI tools of this type thus hold the potential to bestow a warfighting advantage by 
improving the quality of information available to decisionmakers.”° 


Predictability 


AI algorithms often produce unpredictable and unconventional results. In March 2016, the AI 
company DeepMind created a game-playing algorithm called AlphaGo, which defeated a world- 
champion Go player, Lee Sedol, four games to one. After the match, Sedol commented that 
AlphaGo made surprising and innovative moves, and other expert Go players subsequently stated 
that AlphaGo overturned accumulated wisdom on game play.””° AI’s capacity to produce 
similarly unconventional results in a military context may provide an advantage in combat, 
particularly if those results surprise an adversary. 


However, AI systems can fail in unexpected ways, with some analysts characterizing their 
behavior as “brittle and inflexible.” Dr. Arati Prabhakar, the former DARPA Director, 
commented, “When we look at what’s happening with AI, we see something that is very 
powerful, but we also see a technology that is still quite fundamentally limited ... the problem is 
that when it’s wrong, it’s wrong in ways that no human would ever be wrong.”””8 


Al-based image recognition algorithms surpassed human performance in 2010, most recently 
achieving an error rate of 2.5% in contrast to the average human error rate of 5%; however, some 
commonly cited experiments with these systems demonstrate their capacity for failure.” As 
illustrated in Figure 4, researchers combined a picture that an AI system correctly identified as a 
panda with random distortion that the computer labeled “nematode.” The difference in the 
combined image is imperceptible to human eyes, but resulted in the AI system labeling the image 
as a gibbon with 99.3% confidence. 
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Figure 4.Al and Image Classifying Errors 
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In another experiment, an AI system 
described the picture in Figure 5 as “a young 
boy is holding a baseball bat,” demonstrating 
the algorithm’s inability to understand 
context. Some experts warn that AI may be 
operating with different assumptions about 
the environment than human operators, who 
would have little awareness of when the 
system is outside the boundaries of its 
original design.”!° 


Similarly, AI systems may be subject to 
algorithmic bias as a result of their training 
data. For example, researchers have 
repeatedly discovered instances of racial bias 
in AI facial recognition programs due to the 
lack of diversity in the images on which the 
systems were trained, while some natural 
language processing programs have 


Figure 5.Al and Context 
“A Young Boy is Holding a Baseball Bat” 


Source: John Launchbury, “A DARPA Perspective on 
Artificial Intelligence,” https://www.darpa.mil/ 


attachments/AIFull.pdf, p. 23. 


developed gender bias.”'! This could hold significant implications for AI applications in a military 
context, particularly if such biases remain undetected and are incorporated into systems with 


lethal effects. 


“Domain adaptability,” or the ability of AI systems to adjust between two disparate environments, 
may also present challenges for militaries. For example, one AI system developed to recognize 
and understand online text was trained primarily on formal language documents like Wikipedia 


articles. The system was later unable to interpret more informal language in Twitter posts. 


212 


Domain adaptability failures could occur when systems developed in a civilian environment are 


transferred to a combat environment.?!3 
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AI system failures may create a significant risk if the systems are deployed at scale. One analyst 
noted that although humans are not immune from errors, their mistakes are typically made on an 
individual basis, and they tend to be different every time. However, AI systems have the potential 
to fail simultaneously and in the same way, potentially producing large-scale or destructive 
effects.”!* Other unanticipated results may arise when U.S. AI systems interact with adversary AI 
systems trained on different data sets with different design parameters and cultural biases.”!° 


Analysts warn that if militaries rush to field the technology prior to gaining a comprehensive 
understanding of potential hazards, they may incur a “technical debt,” a term that refers to the 
effect of fielding AI systems that have minimal risk individually but compounding collective risk 
due to interactions between systems.”'® This risk could be further exacerbated in the event of an 
AI arms race.?!” 


Explainability 


Further complicating issues of predictability, the types of AI algorithms that have the highest 
performance are currently unable to explain their processes. For example, Google created a cat- 
identification system, which achieved impressive results in identifying cats on YouTube; 
however, none of the system’s developers were able to determine which traits of a cat the system 
was using in its identification process.”!* This lack of so-called “explainability” is common across 
all such AI algorithms. To address this issue, DARPA is conducting a five-year research effort to 
produce explainable AI tools.”!” 


Other research organizations are also attempting to do a backwards analysis of these types of 
algorithms to gain a better understanding of their internal processes. In one such study, 
researchers analyzed a program designed to identify curtains in images and discovered that the AI 
algorithm first looked for a bed rather than a window, at which point it stopped searching the 
image. Researchers later learned that this was because most of the images in the training data set 
that featured curtains were bedrooms.””° The project demonstrated the possibility that training sets 
could inadvertently introduce errors into a system that might not be immediately recognized or 
understood by users. 


Explainability can create additional issues in a military context, because the opacity of AI 
reasoning may cause operators to have either too much or too little confidence in the system. 
Some analysts are particularly concerned that humans may be averse to making a decision based 
entirely on AI analysis if they do not understand how the machine derived the solution. Dawn 
Meyerriecks, Deputy Director for Science and Technology at the CIA, expressed this concern, 
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arguing, “Until AI can show me its homework, it’s not a decision quality product.”””! Increasing 

explainability will thus be key to humans building appropriate levels of trust in AI systems. As a 

U.S Army study of this issue concludes, only “prudent trust” will confer a competitive advantage 
for military organizations.”” 


Additional human-machine interaction issues that may be challenged by insufficient 
explainability in a military context include the following: 


e Goal Alignment. The human and the machine must have a common 
understanding of the objective. As military systems encounter a dynamic 
environment, the goals will change, and the human and the machine must adjust 
simultaneously based on a shared picture of the current environment.” 


e Task Alignment. Humans and machines must understand the boundaries of one 
another’s decision space, especially as goals change. In this process, humans 
must be consummately aware of the machine’s design limitations to guard 
against inappropriate trust in the system.” 


e Human Machine Interface. Due to the requirement for timely decisions in 
many military AI applications, traditional machine interfaces may slow down 
performance, but there must be a way for the human and machine to coordinate 
in real time in order to build trust.””° 


Finally, explainability could challenge the military’s ability to “verify and validate” AI system 
performance prior to fielding. Due to their current lack of an explainable output, AI systems do 
not have an audit trail for the military test community to certify that a system is meeting 
performance standards.”*° DOD is currently developing a framework to test AI system lifecycles 
and building methods for testing AI systems in diverse environments with complex human- 
machine interactions.’ In August 2020, DOD awarded a $106 million contract to Deloitte 
Consulting to build the JAIC’s Joint Common Foundation, which is to “provide an AI 
development environment to test, validate, and field AI capabilities at scale across the 
Department of Defense.” 


In addition, the National Security Commission on Artificial Intelligence is to provide 
recommendations for operationalizing predictability and explainability in AI systems. As the 
interim report notes, “recommendations might include identifying gaps in current processes, 
providing guidance on needed policy and technical standards, and identifying areas where future 
R&D and workforce training is necessary.”””’ 
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Exploitation 


AI systems present unique pathways for 
adversary exploitation. First, the proliferation 
of AI systems will increase the number of 
“hackable things,” including systems that 
carry kinetic energy (e.g., moving vehicles), 
which may in turn allow exploitive actions to 
induce lethal effects. These effects could be 
particularly harmful if an entire class of AI 
systems all have the same exploitable 
vulnerability.”*° 


Figure 6.Adversarial Images 


In addition, AI systems are particularly 
vulnerable to theft by virtue of being almost 
entirely software-based. As one analyst points 
out, the Chinese may be able to steal the plans 
for an F-35, but it will take them years to find 


Source: Evan Ackerman, “Slight Street Sign 
Modifications Can Completely Fool Machine Learning 


the materials and develop the manufacturing Algorithms,” IEEE Spectrum, August 4, 2017, 
processes to build one. In contrast, stolen https://spectrum.ieee.org/cars-that-think/ 
software code can be used immediately and transportation/sensors/slight-street-sign- 
reproduced at will.23! This risk is amplified by modifications-can-fool-machine-learning-algorithms. 


the dual-use nature of the technology and the fact that the AI research community has been 
relatively open to collaboration up to this point. Indeed, numerous AI tools developed for civilian 
use—but that could be adapted for use in weapon systems—have been shared widely on 
unclassified internet sites, making them accessible to major military powers and nonstate actors 
alike. 


Finally, adversaries may be capable of deliberately introducing the kinds of image classification 
and other errors discussed in the “Predictability” section above. In one such case, researchers who 
had access to the training data set and algorithm for an image classifier on a semiautonomous 
vehicle used several pieces of strategically placed tape (as illustrated in Figure 6) to cause the 
system to identify a stop sign as a speed limit sign. In a later research effort, a team at MIT 
successfully tricked an image classifier into thinking that a picture of machine guns was a 
helicopter—without access to the system’s training data or algorithm.”* These vulnerabilities 
highlight the need for robust data security, cybersecurity, and testing and evaluation processes as 
military AI applications are developed. 
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ATs Potential Impact on Combat 


Although AI has not yet entered the combat arena in a serious way, experts are predicting the 
potential impact that AI will have on the future of warfare. This influence will be a function of 
many factors, including the rate of commercial investment, the drive to compete with 
international rivals, the research community’s ability to advance the state of AI capability, the 
military’s general attitude toward AI applications, and the development of Al-specific warfighting 
concepts.7** 


Many experts assert that there is a “sense of inevitability” with AI, arguing that it is bound to be 
substantially influential.” Nevertheless, in January 2016, then-Vice Chairman of the Joint Chiefs 
of Staff General Paul Selva, intimated that it may be too early to tell, pointing out that DOD is 
still evaluating AI’s potential. He stated, “The question we’re trying to pose now is, ‘Do the 
technologies that are being developed in the commercial sector principally provide the kind of 
force multipliers that we got when we combined tactical nuclear weapons or precision and 
stealth?’ If the answer is yes, then we can change the way that we fight.... If not, the military will 
seek to improve its current capabilities slightly to gain an edge over its adversaries.”**° There are 
a range of opinions on AI’s trajectory, and Congress may consider these future scenarios as it 
seeks to influence and conduct oversight of military AI applications. 


Minimal Impact on Combat 


While many analysts admit that military AI technology is in a stage of infancy, it is difficult to 
find an expert who believes that AI will be inconsequential in the long run.”*” However, AI critics 
point to a number of trends that may minimize the technology’s impact. From a technical 
standpoint, there is a potential that the current safety problems with AI will be insurmountable 
and will make AI unsuitable for military applications.”** In addition, there is a chance the 
perceived current inflection point in AI development will instead lead to a plateau. Some experts 
believe that the present family of algorithms will reach its full potential in another 10 years, and 
AI development will not be able to proceed without significant leaps in enabling technologies, 
such as chips with higher power efficiency or advances in quantum computing.” The technology 
has encountered similar roadblocks in the past, resulting in periods called “AI Winters,” during 
which the progress of AI research slowed significantly. 


As discussed earlier, the military’s willingness to fully embrace AI technology may pose another 
constraint. Many academic studies on technological innovation argue that military organizations 
are capable of innovation during wartime, but they characterize the services in peacetime as large, 
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inflexible bureaucracies that are prone to stagnation unless there is a crisis that spurs action.” 
Members of the Defense Innovation Board, composed of CEOs from leading U.S. commercial 
companies, remarked in their most recent report, “DOD does not have an innovation problem, it 
has an innovation adoption problem” with a “preference for small cosmetic steps over actual 
change.” 


Another analysis asserts that AI adoption may be halted by poor expectation management. The 
report asserts that overhyped AI capabilities may cause frustration that will “diminish people’s 
trust and reduce their willingness to use the system in the future.””*? This effect could have a 
significant chilling effect on AI adoption. 


Evolutionary Impact on Combat 


Most analysts believe that AI will at a minimum have significant impact on the conduct of 
warfare. One study describes AI as a “potentially disruptive technology that may create sharp 
discontinuities in the conduct of warfare,” further asserting that the technology may “produce 
dramatic improvements in military effectiveness and combat potential.””” These analysts point to 
research projects to make existing weapon systems and processes faster and more efficient, as 
well as providing a means to cope with the proliferation of data that complicate intelligence 
assessments and decisionmaking. However, these analysts caution that in the near future AI is 
unlikely to advance beyond narrow, task-specific applications that require human oversight.” 


Some AI proponents contend that although humans will be present, their role will be less 
significant, and the technology will make combat “less uncertain and more controllable,” as 
machines are not subject to the emotions that cloud human judgment.” However, critics point to 
the enduring necessity for human presence on the battlefield in some capacity as the principle 
restraining factor that will keep the technology from upending warfare. An academic study of this 
trend argues 


At present, even an AI of tremendous power will not be able to determine outcomes in a 
complex social system, the outcomes are too complex — even without allowing for free will 
by sentient agents.... Strategy that involves humans, no matter that they are assisted by 
modular AI and fight using legions of autonomous robots, will retain its inevitable human 
flavor.?“6 
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Pointing to another constraining factor, analysts warn of the psychological impact that 
autonomous systems will have on an adversary, especially in conflict with cultures that place a 
premium on courage and physical presence. One study on this topic quotes a security expert from 
Qatar who stated, “How you conduct war is important. It gives you dignity or not.” 


In addition, experts highlight that the balance of international AI development will affect the 
magnitude of AI’s influence. As one analyst states, “[T]he most cherished attribute of military 
technology is asymmetry.”” In other words, military organizations seek to develop technological 
applications or warfighting concepts that confer an advantage for which their opponent possesses 
no immediate countermeasure. Indeed, that is the U.S. military’s intent with the current wave of 
technological development as it seeks “an enduring competitive edge that lasts a generation or 
more.”””? For this reason, DOD is concerned that if the United States does not increase the pace 
of AI development and adoption, it will end up with either a symmetrical capability or a 
capability that bestows only a fleeting advantage, as U.S. competitors like China and Russia 
accelerate their own respective military AI programs.” 


The democratization of AI technology will further complicate the U.S. military’s pursuit of an AI 
advantage. As the 2018 National Defense Strategy warns, “The fact that many technological 
developments will come from the commercial sector means that state competitors and nonstate 
actors will also have access to them, a fact that risks eroding the conventional overmatch to which 
our Nation has grown accustomed.””°! In these circumstances, AI could still influence warfighting 
methods, but the technology’s overall impact may be limited if adversaries possess comparable 
capabilities. 


Revolutionary Impact on Combat 


A sizeable contingent of experts believe that AI will have a revolutionary impact on warfare. One 
analysis asserts that AI will induce a “seismic shift on the field of battle” and “fundamentally 
transform the way war is waged.””? The 2018 National Defense Strategy counts AI among a 
group of emerging technologies that will change the character of war, and Frank Hoffman, a 
professor at the National Defense University, takes this a step further, arguing that AI may “alter 
the immutable nature of war.” 


Statements like this imply that AI’s transformative potential is so great that it will challenge long- 
standing, foundational warfighting principles. In addition, members of the Chinese military 
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establishment assert that AI “will lead to a profound military revolution.” Proponents of this 
position point to several common factors when making their case. They argue that the world has 
passed from the Industrial Era of warfare into the Information Era, in which gathering, exploiting, 
and disseminating information will be the most consequential aspect of combat operations. 


In light of this transition, AI’s potential ability to facilitate information superiority and “purge 
combat of uncertainty” will be a decisive wartime advantage, enabling faster and higher-quality 
decisions.” As one study of information era warfare states, “[W]inning in the decision space is 
winning in the battlespace.”*°° Members of this camp argue that AI and autonomous systems will 
gradually distance humans from a direct combat role, and some even forecast a time in which 
humans will make strategic-level decisions while AI systems exclusively plan and act at the 
tactical level. In addition, analysts contend that AI may contest the current preference for quality 
over quantity, challenging industrial era militaries built around a limited number of expensive 
platforms with exquisite capabilities, instead creating a preference for large numbers of adequate, 
less expensive systems.” 


A range of potential consequences flow from the assumptions surrounding AI’s impact on 
warfighting. Some studies point to overwhelmingly positive results, like “near instantaneous 
responses” to adversary operations, “perfectly coordinated action,” and “domination at a time and 
place of our choosing” that will “consistently overmatch the enemy’s capacity to respond. ””58 
However, AI may create an “environment where weapons are too fast, small, numerous, and 
complex for humans to digest ... taking us to a place we may not want to go but are probably 
unable to avoid.””*? In other words, AI systems could accelerate the pace of combat to a point in 
which machine actions surpass the rate of human decisionmaking, potentially resulting in a loss 
of human control in warfare.” 


There is also a possibility that AI systems could induce a state of strategic instability. The speed 
of AI systems may put the defender at an inherent disadvantage, creating an incentive to strike 
first against an adversary with like capability. In addition, placing AI systems capable of 
inherently unpredictable actions in close proximity to an adversary’s systems may result in 
inadvertent escalation or miscalculation.”°! 


Although these forecasts project dramatic change, analysts point out that correctly assessing 
future impacts may be challenging. Historians of technology and warfare emphasize that previous 
technological revolutions are apparent only in hindsight, and the true utility of a new application 
like AI may not be apparent until it has been used in combat.” 
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Nevertheless, given ATs disruptive potential, for better or for worse, it may be incumbent on 
military leaders and Congress to evaluate the implications of military AI developments and 
exercise oversight of emerging AI trends. Congressional actions that affect AI funding, 
acquisitions, norms and standards, and international competition have the potential to 
significantly shape the trajectory of AI development and may be critical to ensuring that advanced 
technologies are in place to support U.S. national security objectives and the continued efficacy 
of the U.S. military. 
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